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ABSTRACT 

The domination of the information processing approach 
has shifted research from problem solving strategies to the structure 
and organization of knowledge that characterizes expertise. The 
purpose of this study was to compare the reasoning processes of 12 
clinicians and AO medical students as they responded to 6 positively 
stated multiple choice items involving patient diagnosis. The 
theoretical framework that guided the study was the information 
processing theory of J. R. Anderson, which sees knowledge acquisition 
as a process in which individuals progress from novices to experts in 
sequential stages over a period of time. Clinician and student 
protocols were analyzed using subsets of behaviors related to the 
item stem and the choice options. The reasoning processes of the 
students and clinicians differed. While clinicians integrated the 
information presented in the stem and activated a hypothesis to 
explain all the information, students restated and focused on pieces 
of information in the stem, and used short connective links to 'relate 
one piece of information to an interpretation or possible answer. The 
differences in behavior appeared consistent with the Anderson 
information processing theory. Since students solve items by heeding 
the alternatives, each alternative presented should be integral to 
the central concept of the item and indicate a knowledge deficit or 
misconception. An appendix presents the multiple choice items. 
(Contains three tables and six references.) (SLD) 
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Clinicians' Perceptions of Medical Students' Reasoning on Multiple Choice Items 

Within the context of medical education, recent investigations of reasoning have been pursued from 
the perspective of developments in cognitive science. The domination of the information processing 
approach has shifted research from problem solving strategies to the structure and organization of 
knowledge that characterizes expertise. The purpose of this study was to compare the reasoning 
processes of 12 clinicians and 40 students as they responded to six positively stated multiple choice items. 

Unsuccessful problem solvers are unable to integrate information and generated many diagnoses. 
Patel, Groen, and Arocha (1990) approached the study of expert reasoning using propositional networks. 
They found that expert clinicians, working within their area of expertise, use a form of forward reasoning, 
moving from facts and observations to diagnoses as they solve problems in their areas of expertise. 
Novices and clinicians working outside their areas of expertise make more use of backward reasoning. 
Experts relied on clinically-based models while novices incorporated pathophysiological data to explain a 
patient's problem. In these investigations subjects with varying levels of education and experience were 
presented with a written description of a patient case as a stimulus. After the case was read, it was 
removed and subjects were asked to recall as much of the text as possible, explain the underlying 
pathophysiology, and provide a diagnosis. 

Viewing the organization of knowledge as a set of syntactic and semantic axes, Lemieux and 
Bondage (1992) showed that successful problem solvers use several semantic axes and that a patient's 
clinical features are organized into clusters that signify deep and broad understanding. 

Few investigations have examined the structure and organization of knowledge using MC items 
as stimulus material. Using a think aloud approach, Skakun, (1994) and Skakun, Maguire, and Cook 
(1994), asked forty third and fourth year medical students to think aloud as they responded to thirty multiple 
choice items. From the problem solving literature and student think aloud protocols, Skakun (1994) 
developed a list of 25 student-item interaction activities. Three categories of activities emerged from the 
list. The first category consisted of five actions that dealt with whether students generated an answer 
before reading the alternatives and with students treatment of the alternatives. Sixteen behaviours related 
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to the disposition of alternatives comprised the second category. The third category consisted of four 
< activities related to successful problem solving. 

Students used one of three broad strategies when responding to the items. In the first strategy, 
students read the stem and then searched the alternatives for the answer. Upon selecting an answer, the 
remaining alternatives were discarded with reasons. In the second strategy, students read the stem, 
generated an answer, and then went on to search the alternatives for the generated answer or a better 
answer. Incorrect options were discarded with reasons. In the third strategy, students activated a 
hypothesis (diagnosis), although the items did not request a diagnosis. Instead, the item requested other 
aspects such as treatment, management, and appropriate investigations. Upon reading the item, students 
activated hypotheses, determined the task, and then went on to search the alternatives for the answer. 
Alternatives were discarded with reasons. 



Theoretical Framework 

The theoretical framework that guided this study was Anderson's (1993; 1995) information 
processing theory (ACT ). According to Anderson, knowledge acquisition is a process in which individuals 
progress from novices (e.g., medical students) to experts (e.g., clinicians) in sequential stages over a period 
of time. In the declarative stage, an individual acquires facts and concepts characterized as schema. With 
further education and experience, schemas are compiled and assimilated into performance schema. This 
allows for rapid retrieval of information (procedural knowledge). The third stage, efficiency, automaticity, 
and metacognition occur when performance is spontaneous and conscious cognition is minimal. 



Method 

The clinicians for this study consisted of four family practitioners and eight internal medicine 
specialists, all with teaching appointments in the Faculty of Medicine. For this study, the think aloud 
protocols of students interacting with six multiple choice items were selected from Skakun (1994). 

Six positively stated MC items were selected from the Skakun, Maguire, and Cook (1994) study. 
The items are listed in Appendix 1. 
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Clinicians thought aloud while solving each MC item. As part of the interview, each participant was 
briefed on the purpose, tasks, and procedures, and given an example of a think aloud protocol. In addition, 
each clinician was given a practice item before the interview began. All interviews were audiotaped, 
transcribed, and verified by listening to the audiotape while reading the protocol for accuracy. 



Analysis 

The clinician and student protocols were analyzed using a subset of the categories reported by 
Skakun, Maguire, and Cook (1994). The eight subsets of behaviours reported in the present paper consist 
of two larger sets of behaviours - those related to the stem and those related to the disposition of the 
options. Coding of the categories in the clinicians' and students' protocols was done independently by two 
individuals, then the results were discussed. Resolution was reached on the coding of the behaviours. 

The subsets of behaviours for those that describe the stem behaviours were of four types. 
Students tended to restate the information in the stem in their own words, paraphrase the scenario 
(restates). Other individuals redescribed or focussed on a key feature and defined it in a different way 
(redescribes). Both of these strategies are examples of backward reasoning and are more typical of novice 
reasoning. Some individuals generated an answer or problem space using the information presented in 
the stem immediately, before reading the alternatives (generates a/s). Others activated a hypothesis using 
the information presented in the stem (activates hyp.). The last two behaviours are examples of forward 
reason, and are more typical of expert reasoning. 

Clinicians and students disposed of options using four strategies. When forward reasoning was 
used, individuals tended to use two complimentary strategies. Individuals would discard an option because 
it would not represent the first course of action, it was inappropriate or less appropriate, unimportant or 
less important (priority). Alternatives were eliminated because the description or data in the stem did not 
match the alternative (description). These two behaviours were interpreted as a holistic approach to 
eliminating the alternatives, characteristic of forward reasoning and expert behaviour. Individuals who used 
backward reasoning tended to use discreet bits of information in the alternatives and try to link them with 
information in the stem. They eliminated alternatives after reading the stem by stating that the option was 
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unlikely, rare, or less likely than another option (likelihood). Individuals tended to eliminate alternatives by 
! elaborating on specific pieces of clinical information presented in the stem, associating discreet bits of 

information. Condition-action pairs (i.e., if-then statements) were commonly used (association). These 
behaviours were typical of individuals who were novices, that is, who used backward reasoning to solve 
an item. They approached solving the item using an atomistic strategy. 

To compare the differences between the clinicians' and students' reasoning behaviour, the following 
procedure was used. First, the clinicians' and students' behaviours were standardized by dividing the 
number of occurrences of each behaviour for each item by the clinicians (12) and students (40) 
respectively. Second, the difference between the clinicians' and students' standardized behaviour ratio was 
computed. Third, values greater than j ±0.20 1 were considered as indicative of practical significance. 

Results 

As shown in Table 1, the items' disciplines and classifications varied as did the clinicians' and 
students' item difficulties. Not all clinicians chose the keyed response to every item. From Table 1 we can 
see that the rank order of difficulties differs between the two groups. Since for all items except one, a 
greater proportion of clinicians chose the keyed option, the difference may be largely accounted for by the 
greater clinical experience amongst the clinician group. 

Tables 2 and 3 present clinicians' and students' approaches to the treatment of the item stems and 
alternatives. Differences between standardized behaviour ratios favouring clinicians are noted with (C), and 
a (S) is used for differences favouring the students. An equals sign (=) appears for no differences. To 
illustrate the behaviours, an example for each category follows. 

With respect to the behaviours related to the stem (Table 2), it appears that students are more 
likely to restate and redescribe the information presented in the stem than clinicians. For example, as 
shown in Table 2, more students restated and redescribed the information in the stem for item 347 than 
did clinicians as shown by the (C) sign. When students restated the information in the stem to 
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Table 1 

Item Information and Clinicians' and Students' Performance 





Item 




Difficulty 


Number 


Discipline 


Classification 


Clinician 


Student 


771 


Gastrointestinal 


Management 


1.00 


0.55 


317 


Gastrointestinal 


Management 


0.92 


0.80 


347 


Pulmonary 


Comprehension 


0.92 


0.53 


582 


Cardiovascular 


Diagnosis 


0.92 


0.75 


733 


Neurology 


Diagnosis 


0.67 


0.45 


332 


Pulmonary 


Diagnosis 


0.42 


0.55 



familiarize themselves with the scenario, they tended to paraphrase the details and facts. For item, 317, 
Student 038 said, 

A 24 year old flight attendant, tired and losing weight in spite of good appetite. Voluminous, pale, 
foul smelling stools. Foul smelling. I think about malabsorptive, like fat. Having bowel difficulty 
in early childhood and being fed diet consisting largely of bananas. I think the bananas because 
she probably have low potassium. That is why she is put on a diet. So she had bowel problems 
since early childhood. So it is malabsorptive confirmed by the fat in the stool. Blood picture 
shows anemia. If there is malabsorption is likely she has anemia. Which of the following diets 
would you select for this patient. 

I guess I'm thinking what causes malabsorptive diarrhea in young females. Celiac disease. 
But, I'm not very sure about the actual stool picture. 
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Table 2 

Stem Related Behaviours 







Behaviour 






ID 


Restates 


Redescribes 


Generates A/S 


Activates Hyp. 


771 


** 


= 


*** 

C 


= 


317 


= 


* 

S 


C 


S 


347 


= 


c 


= 


c 


582 


s 


s 


S 


c 


733 


= 


s 


= 


= 


332 


s 


s 


s 


c 



NOTE: S Student; = Difference | ±0.20 1 ; C Clinician. 

Student 007 read the stem and quickly scanned the options of item 317, and focussed on 
malabsorption, explaining, "Well let's have a look here and see what they say, gluten free, lactose free, low 
fat, low residue, high residue. So all of this sounds like a malabsorption problem confirmed by an abnormal 
small bowel follow through." 

However, when clinicians generated an answer from the stem for item 31 7, they were quick to offer 
an answer, the keyed response. Clinician Cl was particularly succinct saying, 

A 24 year old airline attendant complains of feeling tired and losing weight in spite of a good 

appetite. For the past year she has noticed voluminous, pale, foul smelling stools. She recalls of 
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having been told of having bowel difficulty in early childhood and of being fed a diet largely 
< consisting of bananas. 

Radiological examination discloses an abnormal small bowel follow through. Biochemical 
analysis of the stool shows an increased amount of fat. The blood picture shows anemia. Which 
of the following diets would you select for this patient? 

The answer is one, gluten free. She has celiac sprue. Nothing else fits. 

There was no pause after this clinician read the stem. It was as though the answer were just a 
continuation of the stem. The response was given confidently, with no hesitation that it was the only correct 
answer. 

When reading the stem, students were more likely to generate answers than clinicians, however, 
these did not necessarily match the keyed response or the distracters. This behaviour was typical for item 
733. Student 035 exemplified this approach by thinking aloud, 

56 year old man presents to his doctor with a one month history of intermittent right facial pain. 
So, right away I'm thinking that the differential diagnosis of facial pain could be sinusitis. It could 
be skin or muscle or something like that. It could be shingles. It could be neurological. 

On examination he is found to have a diminished corneal reflex and slight hearing defect 
on the right so I'm thinking neurological already here. 

... So basically I'm down to acoustic neuroma and trigeminal neuralgia, and I'd pick 
acoustic neuroma. Then again, I'm thinking about intermittent right facial pain. I'd still go with 
acoustic neuroma, number four because basically the trigeminal neuralgia can't explain the 
unilateral hearing loss unless that's a red herring which it could be, but number four I'll go with. 

This student began thinking of various differential diagnoses after the first sentence, then focussed on a 
neurological disease. After reviewing the neuroanatomy of the cranium, matching each of the alternatives 
with the stem, went onto choose the keyed response. 
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When reading the stem, the clinicians' dominant behaviour was one of activating hypotheses. For 
four of the six items they were more likely to activate hypotheses than students. In doing so, they produced 
an integrated explanation of the information presented in the stem. In Item 733 Clinician C2 read the entire 
item and chose the keyed response, acoustic neuroma, as though there wasn't any question as to what 
the correct answer was by saying, "Based on the history, based on the physical findings, the best answer 
is number four, acoustic neuroma." When probed as to how the answer was thought of, this clinician said, 

56 year-old man with a month history of pain. Pain. Symptom is pain. On physical examination 
he's got a diminished right corneal reflex. And, remembering the reflex arc of the corneal reflex, 
one would have to then call on some basic science in neuroanatomy to remember what the reflex 
arc for the corneal reflex is and realize that its got to be through the fifth cranial nerve. 

A slight hearing deficit or hearing defect, which has got to involve the eighth cranial nerve. 
Realizing that those nerves are pretty close together in the brain stem, would suggest, and on the 
same side, in usually in this area, the lesion points to the side the lesion is on, so the best answer 
in my mind would be acoustic neuroma. 

This clinician activated a hypothesis, acoustic neuroma, while reading the stem. Again, the response was 
given quickly and confidently with no hesitation. 

A similar behaviour was used in item 582 when Clinician C2 quickly and confidently activated a 
hypothesis from the stem for item 582 by explaining, 

One week after an anterior myocardial infarction, well, what kinds of complications occur after a 
myocardial infarction? You get pulmonary emboli, what else? 

A 55 year-old man complains of severe pain in the left leg. The leg is cool, pale, and 
pulseless. The most likely diagnosis is: 

I don't even have to read the list. He has thrown an embolus from his myocardium to an 
artery in his leg. I don't know if this is femoral or iliac or popliteal. Where in the leg? He has an 
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arterial embolism. Period. Straight forward. 



Students used a different strategy. One student, Student 004 redescribed the scenario, then generated 
an answer from the stem. This student explained, 

First I'd start with reading the question which is, one week after an anterior myocardial infarct a 
55-year-old man complains of severe pain in the left leg. The leg is cool, pale and pulseless. I 
know it was one week after an anterior myocardial infarction. I know the position of the infarction, 
the duration, the time. He's a 55-year-old man. He's got severe pain in the left leg. The leg being 
cool, pale and pulseless. I can tell that there must be something blocking the arterial circulation 
because if it’s a venous blockage you would have a pulse and the leg would be warm. So I would 
look for the most likely diagnosis. I would look for something in the arterial circulation. 

These examples illustrate differences in strategy between the clinicians and students when they 
read the stem. Clinicians tended to offer responses quickly, without hesitation. They did not falter in their 
decisions. Conversely, this behaviour was not predominant in the students. They tended to restate the 
stem and redescribe the scenario presented in the stem. 

With respect to the behaviours elicited for the processing of alternatives (Table 3), clinicians were 
more likely to prioritize and describe the distractors. Clinicians viewed the item as a whole, and went on 
to describe the content of each alternative. Students tended to eliminate options using likelihoods and 
associative links, viewing each fragment of information in the stem and each distractor. One exception to 
this general practice occurred in item 771 , where more students tended to use description than clinicians. 

For item 317, Clinician C3 used both priority and descriptions to eliminate three alternatives 
explaining, 

Low fat. It might help symptomatically, but it's not getting at the major problem. Low residue. 
That's not going to make any difference. High residue, she's got enough diarrhea, you don't want 
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to give her any more. The best answer is number one. 

This clinician described the conditions for each alternative, and qualified eliminating 'low fat' because it was 
not a major problem, then chose 'gluten free' because it was the answer. 



Table 3 

The Clinicians' and Students' Elimination of Alternatives Behaviours 



Item 


Association 


Likelihood 


Description 


Prioritize 


771 


* 

S 


** 


C 


S 


317 


S 


C 


C 


C 


347 


s 


C 


c 


C 


582 


s 


= 


= 


C 


733 


s 


S 


c 


C 


332 


s 


= 


c 


C 



NOTE: S Student; = Difference j ±0.20 1 ; C Clinician. 

Clinician C5 took a similar approach, describing alternatives for item 347. This clinician said, 

Acute pericarditis is more of a (five second pause) chest pain. It might not be pleuritic. It has 
different characteristics, relieved by sitting up. You may get a flu like illness with that as well. 
Pleurodynia, you wouldn't expect such an acute presentation, suddenly seized with pain. 
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When Student 16 eliminated alternatives in item 347, association and likelihood were used, as this 
student explained, 

Mycoplasma pneumonia usually that doesn't give you severe left sided chest pain acutely it comes 
on more insidiously. (Association) 

Pulmonary embolism. Another likely possibility. The pain and her risk factors involved if she has 
been previously hospitalized or injury to her leg veins or contraceptive pill might predispose her to 
pulmonary embolus. So two and three are the most common so far. (Likelihood) 

Clinicians were more likely to prioritized alternatives for elimination for item 332. Clinician C4 used 
priority for the final elimination of the options. This clinician explained, "I think the most likely diagnosis 
would be three or four. Number five would be a close third, then one and two last. What we are seeing 
here is number four, bronchiectasis." Students used likelihoods to dispose of the alternatives. Student 
005 stated, "Hypogammaglobulinemia is totally possible but unlikely. ... Bronchiectasis is likely. Number 
5 cystic fibrosis is very unlikely as they don't mention it at all. The answer is number 4, bronchiectasis." 

These examples illustrate the behaviours used by clinicians and students when they eliminating 
distractors. Clinicians favoured priority and description to eliminate alternatives, consistent with expert 
reasoning. Students tended to use associative links and likelihoods, behaviours more commonly used by 
novice reasoners. 

Conclusions 

The reasoning processes of the clinicians' and students' differed for the six MC items. Clinicians 
integrated the information presented in the stem and activated a hypothesis to explain all the information. 
Conversely, students restated and focussed on pieces of information in the stem, and used short 
connective links to relate one piece of information to an interpretation or possible answer. This approach 
produced many answers. Because clinicians' and students' differed in their behaviours related to the 
processing of information in the stem, they also differed in their behaviours for eliminating options. 



O 




12 



These differences in behaviour appear to be consistent with Anderson's (1993, 1995) ACT theory. 

; The clinicians' behaviour indicated that their knowledge was well structured and integrated. This suggests 

that clinicians were functioning at the procedural level of knowledge acquisition. In contrast, students' 
knowledge was not elaborate and well structured. Partly because of their level of education, and because 
of lack of experience, they appear to be at the declarative stage. 

Educational Importance of the Study 

The findings have implications when using MC items in medical education assessment and 
standard setting procedures. From an assessment perspective, students solve items by heeding each 
alternative, therefore, each alternative should be integral to the central concept of the item and indicate a 
knowledge deficit or misconception. The clinicians' lack of understanding of how students' solve MC items 
directs us to reexamine the construction of MC items, specifically the role of alternatives. 

From a standard setting perspective, the inclusion of alternatives that do not represent errors or 
misconceptions in knowledge may lead to the setting of unreasonable passing scores. The clinicians' 
judgements may not reflect the students' level of knowledge at this phase of training. 
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Appendix 1 - Multiple Choice Items 



ITEM 1: ID. 771 

The irritable bowel syndrome in adults is a 

diagnosis of exclusion. However, when this 

diagnosis is finally made, you should: 

1. tell the patient the symptoms are always 
due to emotional stress. 

2. tell the patient to take tranquilizers when 
symptoms flare. 

3. tell the patient to return for a complete 
reevaluation (x-rays, blood work, etc.,) in 
three months. 

*4. counsel the patient and prescribe 
metamucii and bran. 

5. counsel the patient and prescribe Lomotil 
and Kaopectate. 



ITEM 2: ID. 317 

A 24 year-old airline flight attendant complains of 
feeling tired and losing weight in spite of a good 
appetite. For the past year she has noticed 
voluminous, pale, foul-smelling stools. She recalls 
being told of having bowel difficulty in early 
childhood and of being fed a diet largely consisting 
of bananas. Radiological examination discloses an 
abnormal small bowel follow through. Biochemical 
analysis of the stool shows an increased amount of 
fat. The blood picture shows anemia. Which of 
the following diets would you select for this 
patient? 

*1. Gluten free 

2. Lactose free 

3. Low fat 

4. Low residue 

5. High residue 



ITEM 3: ID. 347 

A previously healthy 27 year-old female is 
suddenly seized with pleuritic pain in the left chest 
and shortness of breath. The most likely cause is: 

1. mycoplasma pneumonia. 

*2. spontaneous pneumothorax. 

3. pulmonary embolism. 

4. acute pericarditis. 

5. pleurodynia. 
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ITEM 4: ID. 582 

One week after an anterior myocardial infarction, a 
55 year-old man complains of severe pain in the 
left leg. The leg is cool, pale, and pulseless. The 
most likely diagnosis is: 

1 . deep venous thrombosis. 

2. ruptured left iliac aneurysm. 

*3. arterial embolism. 

4. A-V fistula. 

5. arterial thrombosis. 



ITEM 5: ID. 733 

A 56 year-old man presents with a month history of 
intermittent right facial pain. On examination he is 
found to have a diminished right corneal reflex and 
a slight hearing defect on the same side. The 
diagnosis is: 

1. right cerebral tumour. 

2. trigeminal neuralgia. 

3. otitis media. 

*4. acoustic neuroma. 

5. multiple sclerosis. 



ITEM 6: ID. 332 

A 28 year-old environmental activist has a history 
of having had pneumonia four times in the past 
twenty years. She has had a productive cough "all 
her life" which is worse in the winter. Physical 
examination reveals dullness, diminished breath 
sounds and numerous crepitations below T3 
bilaterally. Her fingers are clubbed. The most 
likely diagnosis is: 

1. hypogammaglobulinemia. 

2. congenital heart disease. 

3. bronchiolitis obliterans. 

*4. bronchiectasis. 

5. cystic fibrosis. 
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THE CATHOLIC UNIVERSITY OF AMERICA 

Department of Education, O’Boyle Hall 
Washington, DC 20064 
202 319-5120 

February 27, 1996 
Dear AERA Presenter, 

Congratulations on being a presenter at AERA 1 . The ERIC Clearinghouse on Assessment and 
Evaluation invites you to contribute to the ERIC database by providing us with a written copy of 
your presentation. 

Abstracts of papers accepted by ERIC appear in Resources in Education (RIE) and are announced 
to over 5,000 organizations. The inclusion of your work makes it readily available to other 
researchers, provides a permanent archive, and enhances the quality of RIE. Abstracts of your 
contribution will be accessible through the printed and electronic versions of RIE. The paper will 
be available through the microfiche collections that are housed at libraries around the world and 
through the ERIC Document Reproduction Service. 

We are gathering all the papers from the AERA Conference. We will route your paper to the 
appropriate clearinghouse. You will be notified if your paper meets ERIC's criteria for inclusion 
in RIE: contribution to education, timeliness, relevance, methodology, effectiveness of 
presentation, and reproduction quality. 

Please sign the Reproduction Release Form on the back of this letter and include it with two copies 
of your paper. The Release Form gives ERIC permission to make and distribute copies of your 
paper. It does not preclude you from publishing your work. You can drop off the copies of your 
paper and Reproduction Release Form at the ERIC booth (23) or mail to our attention at the 
address below. Please feel free to copy the form for future or additional submissions. 

Mail to: AERA 1996/ERIC Acquisitions 

The Catholic University of America 
O'Boyle Hall, Room 210 
Washington, DC 20064 

This year ERIC/AE is making a Searchable Conference Program available on the AERA web 
page (http : //tikkun . ed . asu . edu/aera/) . Check it out! 

Sincerely, 

FIL 

(^Lawrence M. Rudner, Ph.D. 

Director, ERIC/AE 




'If you are an AERA chair or discussant, please save this form for future use. 
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